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Abstract 

A group of mutually trusting clients outsources a computation service to a remote server, which 
they do not fully trust and that may be subject to attacks. The clients do not communicate with 
each other and would like to verify the correctness of the remote computation and the consistency 
of the server's responses. This paper presents the Commutative-Operation verification Protocol 
(COP) that ensures linearizability when the server is correct and preserves fork-linearizability in 
any other case. Fork-linearizability ensures that all clients that observe each other's operations are 
consistent in the sense that their own operations and those operations of other clients that they see 
are linearizable. COP goes beyond previous protocols in supporting wait-free client operations for 
sequences of commutative operations. 

1 Introduction 

With the advent of cloud computing, most computations run in remote data centers and no longer on 
local devices. As a result, users are bound to trust the service provider for the confidentiality and the 
correctness of their computations. This work addresses the integrity of outsourced data and computa- 
tions and the consistency of the provider's responses. Consider a group of mutually trusting clients who 
want to collaborate on a resource that is provided by a remote minimally trusted server. This could 
be a wiki containing data of a common project, an archival document repository, or a groupware tool 
running in the cloud. A subtle change in the remote computation, whether caused inadvertently by a 
bug or deliberately by a malicious adversary, may result in wrong responses to the clients. Although the 
clients generally trust the provider, they would like to assess the integrity of the computation, to verify 
that responses are correct, and to check that they all get consistent responses. 

In an asynchronous network model without communication among clients such as considered here, 
the server may perform a forking attack and omit the effects of operations by some clients in her re- 
sponses to other clients. Not knowing which operations other clients execute, the latter group cannot 
detect such violations. The best achievable consistency guarantee in this setting is captured by fork- 
linearizability, introduced by Mazieres and Shasha |[T5l for storage systems. Fork-linearizability en- 
sures that whenever the server in her responses to a client C\ has ignored a write operation executed 
by a client C2, then C\ can never again read a value written by C2 afterwards and vice versa. From 
this property, clients can detect server misbehavior from a single inconsistent operation, which is much 
easier than comparing the effects of all past operations one-by-one. 

Several conceptual ||4j [14j |2j [3j and practical advances |[T9l |6l [131 [T71 have recently been made that 
improve consistency checking and verification with fork-linearizability and related notions for remote 
storage and computation. The resulting protocols ensure that when the server is correct, the service is 
linearizable and (ideally) the algorithm is wait-free, that is, every client's operations complete indepen- 
dently of other clients. It has been recognized, however, that read/write conflicts often cause such proto- 
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cols to block; this applies to consistency verification for storage with fork-linearizable semantics llT5l @] 
and for other forking consistency notions J2l|3]. 

In this paper, we go beyond storage services and propose a new protocol for consistency verification 
of remote computations on a Byzantine server, called the Commutative-Operation verification Protocol 
or COP. It supports arbitrary functionalities, exploits commuting operations, and allows clients to oper- 
ate concurrently and without blocking or aborting whenever feasible, while imposing fork-linearizable 
semantics. Through this guarantee Byzantine behavior of the server can be exposed easily. Clients may 
therefore verify the correctness of a service in an end-to-end way. 

Support for wait-free operations is a key feature for collaboration with remote coordination, as 
geographically separated clients may operate with totally different timing characteristics. Consequently, 
previous work has devoted a lot of attention to identifying and avoiding blocking situations lfT5l l4l [TTll . 
For example, read operations in a storage service commute and do not lead to a conflict. On the other 
hand, when a client writes to a data item concurrently with another client reading from the item, the 
reader has to wait until the write operation completes; otherwise, fork-linearizability is not guaranteed. 
If all operations are to proceed without blocking, though, it is necessary to weaken the consistency 
guarantees to fork-* linearizability iflTI or weak fork-linearizability 0, for instance. COP is wait-free 
and never blocks because it aborts non-commuting operations that cannot proceed. Abortable operations 
have been introduced in this context by Majuntke et al. lfl4l . 

The Blind Stone Tablet (BST) protocol [19] supports an encrypted remote database hosted by an 
untrusted server that is accessed by multiple clients. Its consistency checking algorithm allows some 
commuting client operations to proceed concurrently, but only to a limited extent, as we explain below. 
Furthermore, the protocol guarantees fork-linearizability for database state updates, but does not ensure 
it for certain responses output by a client. 

SPORC considers a groupware collaboration service whose operations may not commute, but can 
be made to commute by applying operational transformations. Through this mechanism, different ex- 
ecution orders still converge to the same state. All SPORC operations are wait-free and respect fork-* 
linearizability. 

1.1 Contributions 

This paper considers a generic service executed by an untrusted server and investigates protocols for 
consistency verification through fork-linearizable semantics. It explores the relation between commut- 
ing operations in the service specification and client operations that may proceed concurrently. 

More concretely, this paper introduces the Commutative-Operation verification Protocol (COP) and 
makes three contributions: 

1. COP is the first wait-free protocol that emulates an arbitrary functionality on a Byzantine server 
with fork-linearizability and supports commuting operation sequences. 

2. COP allows clients to proceed at their own speed, regardless of the behavior of other clients, when 
they execute non-conflicting sequences of operations. 

3. We formally prove COP correct and demonstrate that all completed operations and their responses 
respect fork-linearizability. 

COP follows the general pattern of most previous fork-linearizable emulation protocols. For deter- 
mining when to proceed with concurrent operations, it considers sequences of operations that jointly 
commute, in contrast to earlier protocols, which considered only isolated operations. 

In COP, the server merely coordinates client-side operations but does not compute the results. This 
conceptually simple approach can be found in many related protocols (19[ [H |6[ and practical collab- 
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oration systems (gi{j], Mercurial; it also represents the common trend of cloud computing to shift 
computation to the client and coordination to the cloud. 

1.2 Related work 

Storage protocols. Fork-linearizability has been introduced (under the name of fork consistency) to- 
gether with the SUNDR storage system ITT31 ITU1 . Conceptually SUNDR operates on storage objects with 
simple read/write semantics. Subsequent work of Cachin et al. [4 ] improves the efficiency of untrusted 
storage protocols. A lock-free storage protocol with abortable operations, which lets all operations 
complete in the absence of step contention, has been proposed by Majuntke et al. Ifl4l . 

FAUST [3] and Venus 1 17] go beyond the fork-linearizable consistency guarantee and model occa- 
sional message exchanges among the clients. This allows FAUST and Venus to obtain stronger seman- 
tics, in the sense that they eventually reach consistency (in the sense of linearizability) or detect server 
misbehavior. In the model considered here, fork-linearizability is the best possible guarantee ||T5l . 

Blind Stone Tablet (BST). The BST protocol lfT9ll considers transactions on a common database, 
coordinated by the remote server. Clients first simulate a transaction on their own copy, potentially 
generating local output, then coordinate with the server for ordering the transaction. From the server's 
response the client determines if his transaction commutes with other, pending transactions invoked by 
different clients that were reported by the server. If they conflict, the client undoes the transaction and 
basically aborts; otherwise, he commits the transaction and relays it via the server to other clients. When 
a client receives such a relayed transaction, the client applies the transaction to its database copy. 

BST has two limitations: First, because a client applies his own transactions only when all pending 
transactions by other clients have been applied to his own state, state changes induced by his transactions 
are delayed in dependence on other clients. Thus, he cannot always execute his next transaction from 
the modified state and obtain a correct output. Second, the notion of "trace consistency" in the analysis 
of the BST protocol considers only transactions that have been applied to the local state, not local output 
generated by the client. Hence fork-linearizability is not shown for the service responses but only for 
those transactions that clients have applied to their state (the former may occur long before the latter). 

COP is strictly more general than BST, as it allows one client to execute multiple operations inde- 
pendently of the other clients, as long as his sequence of operations jointly commutes with the sequence 
of pending operations by other clients. Note that two operations o\ and 02 may independently commute 
with an operation 03 from a particular starting state, but their concatenation, o\ o 02, may not commute 
with 03. 

Non-blocking protocols. SPORC [6] is a group collaboration system where operations do not need 
to be executed in the same order at every client by virtue of employing operational transforms. The 
latter concept allows to shift operations to a different position in an execution by transforming them 
according to properties of the skipped operations. Differently ordered and transformed variants of a 
common sequence converge to the same end state. 

SPORC achieves fork-* linearizability ITTTI . which is closely related to weak fork-linearizability |f3]; 
both notions are relaxations of fork-linearizability that permit concurrent operations to proceed without 
blocking, such that protocols become wait-free. The increased concurrency is traded for weaker consis- 
tency, as up to one diverging operation may exist between the views of different clients and cannot be 
detected. 
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FAUST 0, mentioned before, never blocks clients and enjoys eventual consistency, but guarantees 
only weak fork-linearizability. 

In contrast to the SPORC and FAUST protocols, COP ensures the stronger fork-linearizability con- 
dition, where every operation is consistent as soon as it completes. SPORC is not weaker nor stronger 
than COP: On one hand, SPORC seems more general as it never blocks clients even for operations that 
do not appear to commute; on the other hand, though, SPORC only supports functions with suitably 
transformable operations and it has no provisions for handling conflicting operations, whereas COP 
works for arbitrary functions. 

In all above protocols for generic services (BST, SPORC, and COP), all clients execute all opera- 
tions. This is not necessary for storage protocols (SUNDR and FAUST) because their operations are 
simpler. 

Last but not least, the protocol of Cachin [1] provides also fork-linearizable execution for generic 
services like COP. However, the approach is inherently blocking and requires the service to satisfy a 
cryptographic notion of "separated authenticated execution." 

1.3 Organization of the paper 

The paper continues by introducing the notation and basic concepts in Section[2] The subsequent section 
presents COP and Section 0] proves that COP emulates an arbitrary functionality on a Byzantine server 
with fork-linearizability. 

2 Definitions 

System model. We consider an asynchronous distributed system with n clients, C\,...,C n and a 
server S, modeled as processes. Each client is connected to the server through an asynchronous, reliable 
communication channel that respects FIFO order. A protocol specifies the operations of the processes. 
All clients are correct and follow the protocol, whereas S operates in one of two modes: either she is 
correct and follows the protocol or she is Byzantine and may deviate arbitrarily from the specification. 

Functionality. We consider a deterministic functionality F (also called a type) defined over a set of 
states S and a set of operations O. F takes as arguments a state s £ S and an operation o E O and 
returns a tuple (s', r), where s' € S is a state that reflects any changes that o caused to s and r € 1Z is a 
response to o: 

(s',r)^F(s,o). 

This is also called the sequential specification of F. 

We extend this notation for executing a sequence of operations (oi, . . . , Ofc), starting from an initial 
state so> and write 

(s',r) = F(s , (oi, . . . ,o k )) 

for (sj, rj) = F(si-i,Oi) with i = 1, . . . , k and (s', r) = F(si~, r^). Note that an operation in O may 
represent a batch of multiple application-level operations. 

Commutative Operations. Commutative operations of F play a role in protocols that may execute 
multiple operations concurrently. Two operations 01,02 6 O are said to commute in a state s if and only 
if these operations, when applied in different orders starting from s, yield the same respective states and 
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responses. Formally, if 



(s', ri ) <- F(s,oi), (s",r 2 ) <- F(s',o 2 ); and 
(t',q 2 ) <- F(a,02), (t",gi) <- F(*',0i) 

then 

ri = gi, r 2 = q 2 , s" = t". 

Furthermore, we say two operations oi,o 2 £ O commute when they commute in any state of S. 

Also sequences of operations can commute. Suppose two sequences p\ and p 2 consisting of opera- 
tions in O are mixed together into one sequence tt such that the partial order among the operations from 
pi and from p 2 is retained in tt, respectively. If executing tt starting from a state s gives the same respec- 
tive responses and the same final state as for every other such mixed sequence, in particular for p\ o p 2 
and for p 2 o pi, where o denotes concatenation, we say that p\ and p 2 commute in state s. Analogously, 
we say that p\ and p 2 commute if they commute in any state. 

Operations that do not commute are said to conflict. Commuting operations have been investigated 
by Weihl [18 ] in the context of concurrency control. We define a Boolean predicate commutep(s, p\ , p 2 ) 
that is true if and only if pi and p 2 commute in s according to F. 

Abortable services. When operations of F conflict, a protocol may either decide to block or to abort. 
Aborting and giving the client a chance to retry the operation at his own rate has often advantages 
compared to blocking, which might delay an application in unexpected ways. 

As in previous work lfl4l . we permit operations to abort and augment F to a functionality F' ac- 
cordingly. F' is defined over the same set of states S and operations O as F, but returns a tuple defined 
over S and 1Z U {J-}. F' may return the same output as F, but F' may also return _L and leave the state 
unchanged, denoting that a client is not able to execute F. Hence, F' is a non-deterministic relation and 
satisfies 

F'(s,o) = {( a ,±), F(s,o)}. 

Since F' is not deterministic, a sequence of operations no longer uniquely determines the resulting state 
and response value. 

Operations and histories. The clients interact with F through operations provided by F. As opera- 
tions take time, they are represented by two events occurring at the client, an invocation and a response. 
A history of an execution a consists of the sequence of invocations and responses of F occurring in a. 
An operation is complete in a history if it has a matching response. 

An operation o precedes another operation d in a sequence of events a, denoted o < a o', whenever o 
completes before o' is invoked in a. A sequence of events tt preserves the real-time order of a history a if 
for every two operations o and d in tt, if o < a d then o < 7T d . Two operations are concurrent if neither 
one of them precedes the other. A sequence of events is sequential if it does not contain concurrent 
operations. For a sequence of events a, the subsequence of a consisting only of events occurring at 
client Ci is denoted by a\c t (we use the symbol | as a projection operator). For some operation o, the 
prefix of a that ends with the last event of o is denoted by a\°. 

An operation o is said to be contained in a sequence of events a, denoted o G a, whenever at least 
one event of o is in a. We often simplify the terminology by exploiting that every sequential sequence 
of events corresponds naturally to a sequence of operations, and that analogously every sequence of 
operations corresponds to a sequential sequence of events. 

An execution is well-formed if the events at each client are alternating invocations and matching 
responses, starting with an invocation. An execution is fair, informally, if it does not halt prematurely 
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when there are still steps to be taken or messages to be delivered (see the standard literature for a formal 
definition [12 ]). We are interested in a protocol where the clients never block each other. Assuming the 
server is correct, then every operation of a client should complete independently of the other clients, and 
only through steps of the client and the server. We call such a protocol wait-free. 

Consistency properties. Clients interact with F via operations. Recall that every operation at a client 
Ci is associated with an invocation and a response event that occurs at CV We say that Cj executes an 
operation between the corresponding invocation and completion events. 

Definition 1 (View). A sequence of events n is called a view of a history a at a client Cj w.r.t. a 
functionality F if: 

1. 7r is a sequential permutation of some subsequence of complete operations in a; 

2. all complete operations executed by Cj appear in 7r; and 

3. 7r satisfies the sequential specification of F. 

Definition 2 (Linearizability (91). A history a is linearizable w.r.t. a functionality F if there exists a 
sequence of events ir such that: 

1. 7r is a view of a at all clients w.r.t. F; and 

2. 7r preserves the real-time order of a. 

Definition 3 (Fork-linearizability [ 15 1). A history a is fork-linearizable w.r.t. a functionality F if for 
each client Cj there exists a sequence of events 7Tj such that: 

1. 7Tj is a view of a at Cj w.r.t. F; 

2. 7Tj preserves real-time order of a; and 

3. for every client Cj and every operation o G 7Tj n 7Tj it holds that 7Tj|° = 7Tj|°. 

Definition 4 (Fork-linearizable Byzantine emulation [4 |). We say that a protocol P for a set of clients 
emulates a functionality F on a Byzantine server S with fork-linearizability if and only if in every fair 
and well-formed execution of P, the sequence of events observed by the clients is fork-linearizable with 
respect to F, and moreover, if S is correct, then the execution is linearizable w.r.t. F. 

Cryptography. In this paper, we make use of several cryptographic primitives, namely hash functions 
and digital signatures. A hash function hash maps a bit string x of arbitrary length to a short, unique 
representation of fixed length. We use a collision-free hash function; this property ensures that it is 
computationally infeasible to produce two different inputs x and x' such that hash(x) = hash(x'). A 
digital signature scheme provides two operations, sign and verify. We parametrize these operations for 
each client Ci as sign i and verify^ The invocation of sign { takes a bit string m as a parameter and returns 
a signature (f> with the response. The verify i operation takes a string m, and a putative signature as 
parameters and returns a Boolean value. It satisfies that verify j(m, <fi) is true for all i and m if and only 
if Ci has executed sign^m) = 4> before. Only Ci may invoke sign^-), but every client and S may 
invoke verify \. 

3 The commutative-operation verification protocol 

The pseudocode of COP for the clients and the server is presented in Figures [Q-O We assume that the 
execution of each client is well-formed and fair. 
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Notation. The function length(a) for a list a denotes the number of elements in a. Several variables 
are dynamic arrays or maps, which associate keys to values. A value is stored in a map H by assigning 
it to a key, denoted H[k] <— v; if no value has been assigned to a key, the map returns _L. Recall that F 1 
is the abortable extension of functionality F. 

Overview. COP adopts the structure of previous protocols that guarantee fork-linearizable seman- 
tics IPT51 [T9l [0. It aims at obtaining a globally consistent order for the operations of all clients, as 
determined by the server. 

When a client Cj invokes an operation o, he sends a INVOKE message to the server S. He expects to 
receive a REPLY message from S telling him about the position of o in the global sequence of operations. 
The message contains the operations that are pending for o, that is, operations that Cj may not yet know 
and that are ordered before o by S. We distinguish between pending-other operations invoked by other 
clients and pending-self operations, which are operations executed by Cj up to o. 

Client Ci then verifies that the data from the server is consistent. In order to ensure fork-linearizability 
for his response values, the client first simulates the pending-self operations and tests if o commutes with 
the pending-other operations. If the test succeeds, he declares o to be successful, executes o, and com- 
putes the response r according to F; otherwise, O is aborted and the response is r = _L. According to 
this, the status of o is either SUCCESS or ABORT. Through these steps the client commits o. Then he 
sends a corresponding COMMIT message to S and outputs r. 

The server records the committed operation and relays it to all clients via a BROADCAST message. 
When the client receives such a broadcasted operation, he verifies that it is consistent with everything 
the server told him so far. If this verification succeeds, we say that the client confirms the operation. If 
the operation's status was SUCCESS, then the client executes it and applies it to his local state. 

Data structures. Every client locally maintains a set of variables during the protocol. The state s € S 
is the result of applying all successful operations, received in BROADCAST messages, to the initial 
state so- Variable c stores the sequence number of the last operation that the client has confirmed. H is 
a map containing a hash chain computed over the global operation sequence as announced by S. The 
contents of H are indexed by the sequence number of the operations, such that entry H[l] is computed as 
hash(H[l — l]\\o\\l\\i) and represents operation o with sequence number I executed by C,;. A variable u 
is set to o whenever the client has invoked an operation o but not yet completed it; otherwise u is _L. 
Variable Z maps the sequence number of every operation that the client has executed himself to the 
status of the operation. 

The server also keeps several variables locally. She stores the invoked operations in a map / and 
the completed operations in a map O, both indexed by the operations' sequence numbers. Variable t 
determines the global sequence number for the invoked operations. Finally, variable b is the sequence 
number of the last broadcasted operation and ensures that S disseminates operations to clients in the 
global order. 

Protocol. When client Ci invokes an operation o, he stores it in u and sends an INVOKE message to 
S containing o, c, and t, a digital signature computed over o and i. In turn, a correct S sends a REPLY 
message with the list u of pending operations; they have a sequence number greater than c. Upon 
receiving a REPLY message, the client checks that uj is consistent with any previously sent operations 
and uses ui to assemble the successful pending-self operations \i and the pending-other operations 7. He 
then determines whether o can be executed or has to be aborted. 

In particular, during the loop in Algorithm [T] for every operation o in u, Ci determines its sequence 
number I and verifies that o was indeed invoked by Cj from the digital signature. He computes the entry 
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Algorithm 1 Commutative-operation verification protocol (client Cj) 



State 

u € O U {!-}: the operation being executed currently or _L if no operation runs, initially ± 
c G No: sequence number of the last operation that has been confirmed, initially 
H : N -> {0, 1}*: hash chain (see text), initially containing only H[0] = NULL 
Z : No — > 2: status map (see text), initially empty 
s£S: current state, after applying operations, initially sq 



upon invocation o do 

ii o 

t sig-iij (invoke II o\\i) 

send message [invoke, o, c, t] to S 



// list of pending-other operations 
// list of successful pending-self operations 



// promised sequence number of o 



II server replies are inconsistent 
// server replies are inconsistent 



upon receiving message [reply, u] from S do 

7<-0 

k <- 1 

while k < length(uj) do 

(o, j,r) <- uj[k] 

I <— C + k 

if not verify -(r, invoke||o|| j) then 
halt 

if #[Z] = _Lthen 

if iJ[Z — 1] =±then 
halt 

#[Z] <-hash(tf[Z-l]||o||Z||.7) 
else if H[l] / hash(H[l - l]||o||i||j) then 
halt 

if j = i A Z[l] = success then 
(i <— fx o (o) 

else if j / i then 
7 <- 7 o (o) 

fc «- k + 1 
iffc = lVo/nVj/i then 

halt // last pending operation must equal the current operation 

(a, r) «— F(s, jj) II compute temporary state with successful pending-self operations 

if commuteF(a, (o),j) then 

(a,r) <- F(a,o) 

Z[l] <- SUCCESS 

else 

r ^— _L 

Z[l\ 4r- ABORT 

<p <- sign i (cOMMlT||u||Z||i7[Z]||Z[/]) 

send message [COMMIT, u, I, H[l], Z[l], <fi] to S 

u <- _L 

return r 



8 



Algorithm 2 Commutative-operation verification protocol (client Cj, continued) 



upon receiving message [BROADCAST, o, q, h, z, <fi,j] from S do 
if not (q = c + 1 and verify^, commit ||o||g||/i|| 2)) then 

halt // server replies are not consistent 

if H[q] = _L then // operation has not been pending at client 

H[q] <- hash(H[q - l]\\o\\q\\j) 
if h ^ H[q] then 

halt // server replies are not consistent, operation is not confirmed 

if z = success then 

(s,r) <- F(s,o) II apply the operation and ignore response 

c <- c+ 1 



Algorithm 3 Commutative-operation verification protocol (server S) 
State 

t G No: sequence number of the last invoked operation, initially 
b G No: sequence number of the last broadcasted operation, initially 
/ : N — > O x N x {0, 1}*: invoked operations (see text), initially empty 
O:N->0x{O,l}*x2x{0, 1}* x N: committed operations (see text), initially empty 



upon receiving message [invoke, o, c, r] from Cj do 

ti-t + 1 
I[t] <- (o,i,r) 

u <-{I[c+l],I[c + 2],...,I[t]) 
send message [reply, oj] to Q 



// include non-committed operations and o 



upon receiving message [COMMIT, o, q, h, z, <p] from C, do 
0[q] <- (o,h,z,4>,i) 

while 0[b + 1] / _L do // broadcast operations ordered by their sequence number 

b^b+1 

(o,h',z',<p',j)^0[b] 

send message [BROADCAST, o, b, h', z', 4>',j] to all clients 
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of o in the hash chain from o itself, I, j, and H[l — 1]. If H[l] = _L, then Cj stores the hash value there. 
Otherwise, if H[l] has already been set, Cj verifies that the hash values are equal; this means that o is 
consistent with the pending operation(s) that S has sent previously with indices up to I. 

If operation o is his own and its saved status in Z[l\ was SUCCESS, then he appends it to /x. The 
client remembers the status of his own operations in Z, since commutep depends on the state and that 
could have changed if he applied operations after committing o. 

Finally, when Cj reaches the end of oj (i.e., when Cj considers o = u), he checks that oj is not empty 
and that it contains o at the last position. He then creates a temporary state a by applying fi to the current 
state s, and tests whether o commutes with the pending-other operations 7 in a. If they do, he records 
the status of o as SUCCESS in Z[l] and computes the response r by executing o on state a. If o does not 
commute with 7, he sets status of o to ABORT and r _L. Then Ci signs o together with its sequence 
number, status, and hash chain entry H[l] and includes all values in the COMMIT message sent to S. 

Upon receiving a COMMIT message for an operation o with a sequence number q, the server records 
its content as 0[q] in the map of committed operations. Then she is supposed to send a BROADCAST 
message containing 0[q] to the clients. She waits with this until she has received COMMIT messages 
for all operations with sequence number less than q and broadcasted them. This ensures that completed 
operations are disseminated in the global order to all clients. Note that this does not forbid clients from 
progressing with their own operations as we explain below. 

In a BROADCAST message received by client Cj, the committed operation is represented by a tuple 
(o, q, h, z, 4>,j). He conducts several verification steps before confirming the operation o and applying 
it to his state s. First, he verifies that the sequence number q is the next operation according to his 
variable c, hence, o follows the global order and the server did not omit any operations. Second, he uses 
the digital signature <fi on the information in the message to verify that the client Cj indeed committed o. 
Lastly, the client computes his own hash-chain entry H[q] for o and confirms that it is equal to the 
hash-chain value h from the message. This ensures that Cj and Cj have received consistent operations 
from S up to o. Once the verification succeeds, the client applies o to his state s only if its status z was 
SUCCESS, that is, when Cj has not aborted o. 

Memory requirements. For saving storage space, the client may garbage-collect entries of H and Z 
with sequence numbers smaller than c. The server can also save space by removing the entries in I and 
O for the operations that she has broadcast. However, if new clients are allowed to enter the protocol, 
the server should keep all operations in O and broadcast them to new clients upon their arrival. 

With the above optimizations the client has to keep only pending operations in H and pending-self 
operations in Z. The same holds for the server: the maximum number of entries stored in I and O is 
proportional to the number of pending operations at any client. 

Communication. Every operation executed by a client requires him to perform one roundtrip to the 
server: send an INVOKE message and receive a REPLY. For every executed operation the server simply 
sends a BROADCAST message. Clients do not communicate with each other in the protocol. However, 
as soon as they do, they benefit from fork-linearizability and can easily discover a forking attack by 
comparing their hash chains. 

Messages invoke, commit and broadcast are all of constant size, while the reply message 
contains the list of pending operations uj. If even one client is slow, then the length of ui for all other 
clients grows proportionally to the number of further operations they are executing. To reduce the size 
of REPLY messages, the client can remember all pending operations received from S, and S can send 
every pending operation only once. 
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Wait-freedom. Every client executing COP can proceed with an operation o for F as long as it does 
not conflict with pending operations of other clients. He outputs the response immediately after receiv- 
ing the REPLY message from S. A conflict arises when o does not commute with the pending operations 
of other clients. In this case, the client aborts o and outputs _L, according to F'. 

It is important that the state used by the client for executing o reflects all of his own operations 
executed so far, even if he has not yet confirmed or applied them to his state because operations of 
other clients have not yet completed. Otherwise, the protocol might violate fork-linearizability. COP 
is wait-free because regardless of whether operation o aborts, the client may proceed executing further 
operations. 

4 Analysis 

Theorem 1. The commutative-operation verification protocol in Figures\1^3\emulates functionality F' 
on a Byzantine server with fork-linearizability. 

We prove this theorem through a sequence of lemmas in the remainder of this section. We start by 
introducing additional notation. 

When a client issues a COMMIT signature for some operation o, we say that he commits o. The 
client's sequence number included in the signature thus becomes the sequence number of o; note that 
with a faulty S, two different operations may be committed with the same sequence number by separate 
clients. 

Lemma 2. If the server is correct, then every history a is linearizable w.r.t. F'. Moreover, if the clients 
execute all operations sequentially, then a is linearizable w.r.t. F. 

Proof. Recall that a consists of invocation and response events. We construct a sequential permutation tt 
of a in terms of the operations associated to the events in a. Note that a client sends an INVOKE message 
with his operation to the server, the server assigns a sequence number to the operation and sends it 
back. The client then computes the response and sends a signed COMMIT message to S, containing the 
operation and its sequence number. Since each executed operation appears in a in terms of its invocation 
and response events, tt contains all operations of all clients. 

We order it by the sequence number of the operations. If the server is correct she processes INVOKE 
messages in the order they are received and assigns sequence numbers accordingly. This implies that 
if an operation d is invoked after an operation o completes, then the sequence number of d is higher 
than o's. Hence, tt preserves the real-time order of a. 

We now use induction on the operations in it to show that tt satisfies the sequential specification 
of F'. Note that F' requires a bit of care, as it is not deterministic. For a sequence ui of operations 
of F' in an actual execution, we write successful(io) for the subsequence whose status was SUCCESS; 
restricted to such operations, F' is deterministic. In particular, consider some operation o £ n, executed 
by client Cj. We want to show that d computes (s',r) such that (s',r) G F'(so, successful(ir\°)), 
whereby it outputs r after committing o and stores s' in its variable s after applying o. 

Consider the base case where o is the first operation in tt. Note that S has not reported any pending 
operations to C\ because o is the first operation. Thus, C{ determines that the status of o is SUCCESS, 
computes (s',r) <— F(sq,o) and outputs r. Hence, F' is satisfied. When C\ later receives o in the 
BROADCAST message from S with sequence number 1, the state is also updated correctly. 

Now consider the case when o is not the first operation in tt and assume that the induction assumption 
holds for an operation that appears in tt before o. If the status of o is ABORT, then the client does not 
invoke F, returns _L, and leaves the state unchanged upon applying o. The claim follows. 
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Otherwise, we need to show that the response r/1 and the state s' after applying o satisfy (V , r) = 
F(so, successful(ir\°)). Since S is correct, she assigns unique sequence numbers to the operations. We 
split the operations with a sequence number smaller than that of o in three groups: a sequence p of 
operations that Cj has confirmed before he committed o, this sequence is in the order in which Cj 
confirmed these operations; a sequence 5 of operations of other clients that were reported by S as 
pending to Cj when executing o, ordered as in the REPLY message; and a sequence v of operations that 
Ci has committed itself before o but not yet confirmed or applied, ordered by their sequence number. 

Observe that Cj computes r starting from its own copy of the state s that results after applying all op- 
erations in successful(p) . From the induction assumption, it follows that (s, •) = F(sq, successful(p)) 
because p is a prefix of it. From variable oj in the REPLY message, Cj computes the pending-other 
operations 7 and the successful pending-self operations p. Note that 7 = S and p = successful^) 
as the server is correct. The client computes a temporary state (a,-) = F(s,p). Because o does 
not abort, Ci has determined that o commutes with 7 in a and computed (-,r) = F(a,o). By the 
definition of commuting operation sequences, we have that (s ; , r) = F(a, successful^) o o) and 
(s',r) = F(s, successful^)) since the order of operations in p and 7 is preserved in oj. Hence, 
(s',r) = F(sq, successful(ir\ )). 

The sequence ir preserves the real-time order of a and satisfies the three conditions of a view of a 
at every client Cj w.r.t. F' , hence, a is linearizable w.r.t. F' . 

The second part of the lemma claims that if clients execute operations sequentially, then no client 
outputs _L. Since the sequence of events at every client is well-formed, a client does not invoke an 
operation before he has completed the previous one. Moreover, if clients execute operations sequentially 
then no client invokes an operation while there is a client who has not completed his operation. Hence, 
the server never includes any pending operations in oj of the REPLY message. The check for conflicts is 
never positive, and all operations have status SUCCESS. Hence, no client returns _L and a satisfies the 
sequential specification of F. □ 

The promised view of an operation. Suppose a client Cj executes and thereby commits an opera- 
tion o. We define the promised view to Ci of o as the sequence of all operations that Cj has confirmed 
before committing o, concatenated with the sequence oj of pending operations received in the REPLY 
message during the execution of o, including o itself (according to the protocol Cj verifies that the last 
operation in oj is o). 

Lemma 3. If Cj has confirmed some operation o that was committed by a client Ci, then the sequence 
of operations that Cj has confirmed up to (and including) o is equal to the promised view to Ci of o In 
particular, 

1. if Ci and Cj have confirmed an operation o, then they have both confirmed the same sequence of 
operations up to o; and 

2. the promised view to Ci of o contains all operations executed by Ci up to o. 

Proof. Note that every client computes a hash chain H in which every defined entry contains a hash 
value that represents a sequence of operations. More precisely, if Cj commits o with sequence number I, 
then he has set H[l] <— hash(H[l — l]||o||Z||«); this step recursively defines the sequence represented by 
H[l] as the sequence represented by H[l — 1] followed by o. According to the collision-resistance of the 
hash function, no two different operation sequences are represented by the same hash value. Note that 
no client ever overwrites an entry of H; moreover, if a client arrives at a point in the protocol where he 
might assign some value h to entry H[l] but H[l] 7^ _L, then he verifies that H[l] = h and aborts if this 
fails. 
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Consider the moment when Cj receives the REPLY message during the execution of o. The view of 
o promised to Cj contains the sequence of operations that Cj has confirmed, followed by the list uj in 
the REPLY message, including o. 

For every pending operation p € u, client Cj checks if he has already an entry in H at index I, which 
is the promised sequence number of p to Cj according to w. If there is no such entry, he computes the 
hash value H[l] as above. Otherwise, Cj must have received an operation for sequence number I earlier, 
and so he verifies that o is the same pending operation as received before. Moreover, Cj verifies that his 
last invoked operation is also returned to him as pending and adds it to H. Hence, the new hash value h 
stored in H at the sequence number of o represents the promised view to Cj of o. 

Subsequently, Cj signs o and h together and sends it to the server. Client Cj receives it in a BROAD- 
CAST message from S, to be confirmed and applied with sequence number q. Because Cj verifies the 
signature of Cj on o, q, and h, the hash value h received by Cj represents the promised view to Cj of 
o. Before Cj applies o as his q-th operation, according to the protocol he must have already confirmed 
q — 1 operations one by one. Client Cj also verifies that he has either already computed the same 
H[q) = h or he computes H[q] from his value H[q — 1] and checks H[q] = h. As H[q] represents the 
sequence of operations that Cj has confirmed up to o, from the collision resistance of the hash function, 
this establishes the main statement of the lemma. 

The first additional claim follows simply by noticing that the statement of the lemma holds for i = j. 
For showing the second additional claim, we note that if Cj confirms an operation of himself, then he 
has previously executed it (successful or not). There may be additional operations that Cj has executed 
but not yet confirmed, but Cj has verified according to the above argument that these were all contained 
in uj from the REPLY message. Thus they are also in the promised view of o. □ 

The view of a client. We construct a sequence 7Tj from a as follows. Let o be the operation committed 
by Cj which has the highest sequence number among those operations of Cj that have been confirmed 
by some client C& (including Cj). Define oq to be the sequence of operations confirmed by Cfc up to 
and including o. Furthermore, let /3, be the sequence of operations committed by Cj with a sequence 
number higher than that of o. Then 7Tj is the concatenation of «j and fy. Observe that by definition, no 
client has confirmed operations from 

Lemma 4. The sequence 7Tj is a view of a at Ci w.r.t. F'. 

Proof. Note that 7Tj is defined through a sequence of operations that are contained in a. Hence 7Tj is 
sequential by construction. 

We now argue that all operations executed by Cj are included in 7Tj. Recall that 7Tj = «j o and 
consider o, the last operation in ctj. As o has been confirmed by C&, Lemma [3] shows that on is equal 
to the promised view to Cj of o and, furthermore, that it contains all operations that Cj has executed 
up to o. By construction of 7Tj all other operations executed by Cj are contained in and the property 
follows. 

The last property of a view requires that 7Tj satisfies the sequential specification of F'. Note that F' 
is not deterministic and some responses might be _L. But when we ensure that two operation sequences 
of F' have responses equal to J_ in exactly the same positions, then we can conclude that two equal 
operation sequences give the same resulting state and responses from the fact that F is deterministic. 

We first address the operations in a,. Consider again o, the last operation in Qj, which has been 
confirmed by C&. For the point in time when Cj executes o, define p to be the sequence of operations that 
Cj has confirmed prior to this and define s as the resulting state from applying the successful operations 
in p, as stored in variable s; furthermore, let uj be the pending operations contained in the REPLY 
message from S. Observe that uj can be partitioned in the pending-other operations 7, the successful 
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pending-self operations of Cj as stored in p, the aborted pending-self operations of Cj, and o. Client Cj 
computes the response r for o in state a that results from F(s, p). Before executing o, Cj verifies that o 
commutes with 7 in a. Note that when C, committed some operation p G he has also verified that p 
commuted with the pending-other operations in lo\ p . Hence, the response resulting from executing the 
operations in p followed by p o o is the same as that resulting from executing p o successful^) o o after p 
(recall the notation successful(-) from Lemma |2). Since ui preserves the order of operations in p and 
7, the response is also the same after the execution of p o successful(uj) . Moreover, the state resulting 
from executing the operations in p followed by p o successful(-j) o o is the same as that resulting from 
executing p o successful(Lo). Since p o ui is the promised view to Cj of o, and since Ck has confirmed o, 
Lemma [3] now implies that p o uj is equal to c^. 

To conclude the argument, we only have to show that the abort status for all operations in the 
sequences is the same. Then they will produce the same responses and the same final state. Note 
that when Cj executes some operation o he either computes a response according to F or aborts the 
operation, declaring its status to be SUCCESS or ABORT, respectively. For operations in p this is clear 
from the protocol as the status is included in the BROADCAST message. And whenever Cj later obtains 
o again as a pending-self operation in uj at some index I, he verifies that it is the same operation as 
previously at index I and applies or skips it as before according to the status remembered in Z[l]. Hence, 
the responses of Cj from executing the operations in ctj respect the specification of F' . 

The remainder of tti consists of (3i, whose operations Cj executes himself using F'. Hence, 7Tj 
satisfies the sequential specification of F'. □ 

Lemma 5. If some client Ck confirms an operation o\ before an operation 0% then 02 does not precede 
o± in the execution history a. 

Proof. Let 5k denote the sequence of operations that Ck has confirmed up to 02. According to the 
protocol logic, 5k contains o\, and o\ has a smaller sequence number than 02. Lemma[3] shows that 5k 
is equal to the promised view to Ck of 02, hence, o\ is in the promised view to Ck of 02. Recall that the 
promised view contains operations that have been committed or are pending for other clients. Hence, o\ 
has been invoked before 02 completed. □ 

Lemma 6. The sequence tti preserves the real-time order of a. 

Proof. Recall that 7Tj = aj o and consider first those operations of tti that appear in «j, that is, they 
have been confirmed by some client Ck- Lemma [5] shows that these operations preserve the real-time 
order of a. Second, the operations in (3i are ordered according to their sequence number and they were 
committed by Cj. According to the protocol, Cj executes only one operation at a time and always 
assigns a sequence number that is higher than the previous one. Hence, Pi also preserves the real-time 
order of a. 

We are left to show that no operation in precedes an operation from a{ in a. Recall that a,; is the 
promised view to Q of o (the last operation in ai) and includes the operations that C,; has confirmed or 
received as pending from S after Cj invoked o. Since o precedes all operations from /3j, it follows that 
no operation in ai precedes an operation from /3«. □ 

Lemma 7. If o € 11 i PI ttj then 7Tj|° = 

Proof. As 7Tj = a*i o and ttj = aj o /3j, we need to consider four cases to analyze all operations that 
can appear in tti n nj and the rest are symmetrical. 

1. oGOj and o E af This case happens when (a) C{ and Cj both confirmed o, or when (b) Ci has 
confirmed an operation of Cj or vice versa, or when (c) a client Ck has confirmed operations of 
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Ci and Cj. For (a) and (b) Lemma [3] shows that ai\° = ay|°. In case (c) neither C% nor Cj has 
confirmed o, but o is in their views because Ck has confirmed pending operations of C and Cj . 
Hence, 7Tfc|° = and 7Tfc|° = aj\° again from Lemma [3] 

2. o € /3j and o £ ctj: This case cannot happen, since no client has confirmed operations from by 
definition. 

3. o € ai and o € fif Analogous to the case above. 

4. o € Pi and o G /3j • : This case cannot happen since and /3j contain only pending-self operations 
of Cj and Cj, correspondingly. 

□ 



5 Conclusion 

This paper has introduced the Commutative-Operation verification Protocol (COP), which allows a 
group of clients to execute a generic service coordinated by a remote untrusted server. COP ensures 
fork-linearizability and allows clients to easily verify the consistency and integrity of the service re- 
sponses. In contrast to previous work, COP is wait-free and supports commuting operation sequences, 
but may sometimes abort conflicting operations. 

Given the popularity of outsourced computation and the cloud-computing model, the problem of 
checking the integrity of remote computations has received a lot of attention recently |5] [16). But 
such cryptographic protocols typically address only a two-party model with a single client. Combining 
them with COP or other protocols that guarantee fork-linearizability will represent an important step 
toward a comprehensive consistency-verification solutions for realistic distributed systems. 
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